workaround random SIGSEGV on macOS aarch64 CI
#5757
Conversation
Separate a `let` block into multiple `let` statements to reduce probability of hitting random `SIGSEGV` during flaky CI tests. whatever... 🤯
|
workaround runs here in endless loop until fail: https://ci.status.im/job/nimbus-eth2/job/platforms/job/macos/job/aarch64/job/dev%252Fetan%252Fzz-perpetual/ Seems to be more stable than without it (~10 successful runs so far, none failed) |
| # Until a shared pattern can be identified, it is better to apply | ||
| # workarounds that at least avoid the known to be reachable triggers. | ||
| # The solution is hacky and far from desirable; it is what it is. | ||
| let |
There was a problem hiding this comment.
have you compared the generated C code?
There was a problem hiding this comment.
the extra let in this PR introduces an additional state to the closure iterator code. it is still roughly 60 states for this proc, and no I have no clue why it works.
For the other one I refer to here, no, haven't checked, but tried randomly reducing complexity by splitting among statements and using a temporary variable there made it work for Wasm. But as it's only on experimental project, have not followed up on it. But it also points to inline results unwrapping being a possible trigger for random bugs.
There was a problem hiding this comment.
the extra let in this PR introduces an additional state t
this is an important thing to add to the doc comment - the change in size of the environment changes a number of things such as alignment and which parts of an environment contain pointers and so on, which in turn may have surprising behavioural effects, ie most likely this extra state masks some underlying issue.
There was a problem hiding this comment.
line 1225 already includes that remark
and this is not an artefact of the time of the day (load on server) or something? |
|
There are 50 successful runs in a row with the extra You can verify on |
|
I have removed the extra 
So, yes, this is most likely not time of day dependent. |
The weird `let` bug from #5757 appeared again :-) Document findings.
The weird `let` bug from #5757 appeared again :-) Document findings.
Separate a
letblock into multipleletstatements to reduce probability of hitting randomSIGSEGVduring flaky CI tests.whatever... 🤯